A Mail Client Plugin for Privacy-Preserving Spam Filter Evaluation

We describe a plugin extension to the Thunderbird Mail Client to support standardized evaluation of multiple spam filters on private mail streams. Researchers need not view or handle the subject users’ messages and subject users need not be familiar with spam filter evaluation methodology. All that is required of the user is to install the plugin as a standard extension and to run it on his or her mailbox. The plugin evaluates a spam filter, assuming the user’s existing classification to be accurate, and sends summary results only to the researcher, after allowing the user to verify exactly what is sent. This plugin addresses an outstanding challenge in spam filter evaluation: that of using a broad base of realistic data while satisfying personal and legislative privacy requirements. Previous efforts have used public data which may not be representative, captured data which may be insufficiently private, and obfuscation techniques which compromise the integrity of the data and may also be insufficiently private. We show preliminary results using the tool to evaluate some filters previously evaluated at TREC.